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ABSTRACT 

Nearly half of students who enter college do not graduate. The majority of 
efforts to increase college completion have focused on supporting students 
before or soon after they enter college, yet many students drop out after 
making significant progress towards their degree. In this paper, we report 
results from a multi-year, large-scale experimental intervention conducted 
across five states and 20 broad-access, public colleges and universities to 
support students who are late in their college career but still at risk of not 
graduating. The intervention provided these “near-completer” students with 
personalized text messages that encouraged them to connect with campus- 
based academic and financial resources, reminded them of upcoming and 
important deadlines, and invited them to engage (via text) with campus-based 
advisors. We find little evidence that the message campaign affected academic 
performance or attainment in either the full sample or within individual higher 
education systems or student subgroups. The findings suggest low-cost nudge 
interventions may be insufficient for addressing barriers to completion among 
students who have made considerable academic progress. 
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Colloquium at the Harvard Graduate School of Education, and members of the College Completion Network 
for helpful comments and suggestions on earlier versions of this paper. All errors, omissions, and conclusions 
are our own. 


1. Introduction 

College enrollment rates have increased steadily over the last several decades, yet the 
probability of degree attainment among enrollees has stagnated with just over half of students who 
start college complete within six years of entry (Bound, Lovenheim, & Turner, 2010; Shapiro et 
al., 2016). Low-income students and students of color are significantly less likely to graduate than 
their high-income and white peers; these disparities have only widened over time (Bailey & 
Dynarski, 2011; Chetty, Friedman, Saez, Turner, & Yagan, 2017). 

To date, most efforts to increase college completion rates have focused on supporting 
students before or soon after they enter college. For example, several interventions have focused 
on encouraging students to attend higher-quality colleges from which they are more likely to 
graduate; supporting students to apply for federal student aid, and helping students overcome 
procedural obstacles to matriculation that arise before students arrive on campus (Barr and 
Castleman, 2021; Bettinger, Long, Oreopoulos, & Sanbonmatsu, 2012; Castleman & Page, 2015; 
Hoxby & Turmer, 2013). Colleges and universities have also devoted considerable attention to 
students’ first-year experiences in colleges, with interventions ranging from structured learning 
and advising supports (e.g. CUNY ASAP), learning communities, and first-year seminars, to 
improving remediation policies for students who enter college academically underprepared 
(Bettinger & Long, 2009; Culver and Bowman, 2020; Martorell & McFarlin, 2011; Schnell and 
Doetkott, 2003; Scott-Clayton, Crosta, & Belfield, 2014; Scrivener and Weiss, 2014; Weiss, 
Weissman, Rudd, and Wathington, 2012). 

Evidence suggests that these strategies can increase the share of students that successfully 
navigate the transition to college and make progress towards their degree. However, many students 


who persist beyond the first year of college remain at substantial risk of withdrawing prior to 


earning their degree. More than 40 percent of college students who do not graduate leave after 
their second year of college (Bowen, Chingos, & McPherson, 2009; Shapiro et al., 2014). Recent 
evidence also suggests that one in three dropouts complete at least three-quarters of the credits 
typically required to graduate before they withdraw (Mabel & Britton, 2018). Across the country 
this translates into approximately 400,000 students per college entry cohort who have earned 
substantial credits but do not have a degree to show for it.! 

A combination of limited support for more advanced students and novel challenges that 
arise as students approach completion contribute to these high rates of late withdrawal. The road 
to completion becomes increasingly self-directed as structured student support services taper off 
after the first year of college (Scott-Clayton, 2015). Students may therefore struggle to make and 
follow through on complicated decisions, such as determining which courses to take to fulfill their 
degree requirements, when academic advising is limited and often difficult to access. The non- 
monetary costs of navigating a challenging environment alone may also be difficult for older 
students who lead busy lives and have limited networks of academic support outside of school. 

In this paper, we present experimental evidence from a large scale intervention, called 
Nudges to the Finish Line (N2FL), that we designed in close partnership with 20 colleges and 
universities to increase completion among students who had made significant progress towards 


their degree and were still actively enrolled in college, but who remained at risk of dropout.? We 


' These estimates are based on results from Mabel and Britton (2018), who find that 14 percent of all degree-seeking 
students attending public colleges in Florida and Ohio completed three-quarters of the credits typically required for 
graduation but did not earn an associates or bachelor’s degree. On average those students enrolled in college for 3.2 
years and paid $11,500 per year in out-of-pocket expenses (Horn & Paslov, 2014). Nationwide, state appropriations 
and grants also subsidize the cost of attending public colleges and universities by $10,000 per year on average 
(Schneider, 2010). Of the 15.5 million students enrolled in degree-seeking programs in the United States, this equates 
to approximately 2.2 million students who have earned substantial credits but no degree with substantial costs to 
individuals and to taxpayers. 

> Students were eligible to participate if they had completed at least half of the credits typically required for associate 
or bachelor’s degree attainment at two- and four-year colleges, respectively. 


implemented N2FL in partnership with public higher education institutions in New York City, 
Virginia, Texas, Ohio, and Washington State during the 2016-17 through 2018-19 school years. 
We designed N2FL as a text message campaign that: (1) encouraged students to connect with 
campus-based academic and financial resources; (2) reminded them of upcoming and important 
deadlines; and (3) invited students to engage via text with dedicated college advising staff.* 
Students received approximately one message per week over the course of 2-3 semesters. The 
study sample includes 21,533 students across the 20 partner institutions. 

Several recent papers have found null impacts from large-scale nudge campaigns that 
aimed to improve postsecondary outcomes (Avery et al., 2020; Bird et al., 2021; Gurantz et al., 
2019; Page et al., 2019). The design of N2FL differed in important ways from these studies, which 
led us and partners to believe the intervention could effectively support higher rates of degree 
attainment among students with substantial credits. First, in order to foster trust and perceived 
legitimacy among students, we designed the campaign so that all messages were delivered by a 
specific advisor at students’ college or university.4 Second, the messages actively encouraged 
personal engagement and interaction (via text) between students and advisors; earlier text-based 
nudge campaigns that found positive impacts on students incorporated this interactive feature 
(Castleman and Page, 2015; Castleman and Page, 2016; Oreopoulos and Petronijevic, 2019). 
Third, by virtue of advanced students having less access to support than high school students or 
students early in college, we expected N2FL outreach to provide a more pronounced treatment 


contrast. Finally, to ensure the content was relevant to students at each college or university, we 


3 Students did not have access to this type of text-based advising at two of the 20 institutions. 
4 This stands in contrast to recent ineffective nudge campaigns where messages came from a state or national 
organization with whom students had at best a tenuous relationship. 


worked closely with advising staff at each institution to customize the message content, 
sequencing, and frequency of outreach to their institutional context. 

Nonetheless, results from a multi-cohort randomized trial of N2FL suggest that text-based 
nudges are not effective at addressing the barriers to completion experienced by students who have 
made substantial progress towards a degree. We find little evidence of effects on academic 
performance or attainment in the full sample and across colleges. Our statistical power is such that 
we can reject effects of any significant magnitude. We also find no evidence of varying impacts 
of the N2FL nudges based on students’ baseline predicted probability of dropout prior to earning 
a degree. We analyze numerous dimensions of treatment fidelity and the institutional context to 
investigate why N2FL may not have been effective. For instance, we explore whether impacts vary 
based on the rate at which college advisors responded to students’ texts. We also investigate 
whether N2FL was differentially effective based on whether the college had parallel texting 
campaigns. None of these analyses reveals institutional contexts, advisor practices, or other 
dimensions of project implementation that are associated with heterogeneous treatment impacts. 

Our paper makes several important contributions. Ours is the first paper of which we are 
aware to investigate whether interactive, text-based nudges can improve attainment among 
students who have made substantial progress towards a degree and who are still in college. Several 
interventions have attempted to increase re-enrollment and success among students with 
substantial credits who had already withdrawn, with limited efficacy (Adelman, 2013; Ortagus, 
Tanner, & McFarlin, 2020).° Second, our paper shows that the limited efficacy of nudges in 


postsecondary education is not a function of the level of implementation or the lack of access to 


> For example, through Project Win-Win, a partnership between the Institute for Higher Education Policy and the 
State Higher Education Executive Officers, sixty postsecondary institutions attempted to re-engage former college- 
goers requiring 9 or fewer credits to earn an associate degree (IHEP, 2013). 


text-based advising, as prior papers have hypothesized (e.g. Bird et al., 2021). We find null impacts 
even though the nudges were sent by colleges and universities with whom students had a direct 
connection and invited students to connect with college advisors via text. Finally, by leveraging 
detailed data on the institutional context in which N2FL took place and data on treatment 
implementation and fidelity, we are able to investigate more deeply than prior papers factors that 
could contribute to the efficacy of nudge interventions in higher education. 

The remainder of this paper is structured as follows. In Section 2, we provide a brief 
discussion of the obstacles to completion that disadvantaged populations face at broad access 
institutions and elaborate on which barriers the N2FL intervention is designed to address. In 
Section 3, we present details on the research design, including the participating schools, 
intervention components, study sample, randomization procedure, and empirical strategy. We 
present results in Section 4 and conclude in Section 5 with a discussion of our findings and their 


implications. 


2. Obstacles to College Completion 

A large body of evidence suggests that the costs to completing college are steep and may 
increase as students progress through school. Many students experience high time and effort costs 
to completion because they enter college academically unprepared (Bettinger, Boatman, & Long, 
2013). Resource constraints at broad-access public institutions in the United States, where the 
majority of postsecondary students attend, have escalated those costs by creating a shortage of 
student supports at many institutions (Bound et al., 2010; Deming & Walters, 2017). 

Resource deficiencies are an especially large impediment to student progress because the 
college environment at most broad-access institutions is complicated and difficult to navigate. For 


example, the volume of courses offered at open-enrollment institutions and the array of program 


requirements make it hard for students to know which courses to take in a given term to make 
efficient academic progress (Nodine, Jaeger, Venezia, & Bracco, 2012; Schneider & Yin, 2011). 
With student-to-counselor ratios frequently exceeding 1,000:1, advising is also extremely limited, 
and institutional bureaucracies make it hard for students to access individualized assistance 
(Grubb, 2006; Scott-Clayton, 2015). According to survey research, one-third of community 
college students never use academic advising as a result, even though nearly half of students do 
not understand their graduation requirements or what courses count towards their degree (Center 
for Community College Student Engagement, 2015; Rosenbaum, Deil-Amen, & Person, 2006). 

Within this isolated and confusing landscape, several studies find large effects from 
interventions that provide students entering college with enhanced mentoring, tutoring, and other 
supports (Angrist et al., 2009; Bettinger & Baker, 2014; Castleman & Page, 2016; Clotfelter, 
Hemelt, & Ladd, 2016; Scrivener et al., 2015). However, because these supports are costly, 
institutions typically target resources to first-year students and the impacts of early interventions 
fade out over time (Rutschow, Cullinan, & Welbeck, 2012; Visher, Weiss, Weissman, Rudd, & 
Wathington, 2012). Completing complex tasks may therefore remain a formidable barrier for 
students as they continue to progress in school. 

Furthermore, as students age and take on more responsibilities outside of school (Erisman 
& Steele, 2015; U.S. Department of Education, 2017), the attention to devote to difficult tasks may 
become increasingly limited and lead to more frequent oversight of important deadlines and higher 
psychic costs (e.g., mounting stress, anxiety, and impatience) when obstacles arise. All of these 
factors may contribute to short-sighted perceptions that the immediate costs to continuation exceed 


the unrealized future benefits of earning a degree (Cadena & Keys, 2015; Gurantz, 2015).° These 


® To inform our intervention design, Persistence Plus also conducted student focus groups at each institution 
participating in the pilot year during spring and summer 2016. The most common challenges students identified in 


7 


factors also suggest that targeted interventions may be a cost-effective investment towards 
increasing degree attainment for students on the margin of completing college. On the other hand, 
if the costs to completion for late-stage students are primarily due to other factors, such as 
academic skill deficiencies that make it difficult for students to pass specific course requirements 
in their major, then nudge interventions may have little impact on academic progress and motivate 


the need for more resource-intensive strategies to lower rates of late departure. 


3. Research Design 

We partnered with a diverse array of broad-access, public two- and four-year institutions 
across the country to implement N2FL. All our partner institutions accept 75 percent or more of 
the applicants that apply. Sixty percent of students attending our partner institutions enrolled part- 
time, 32 percent received federal Pell Grants, and 50 percent were students of color. The average 
graduation rate within 150 percent of the expected time (e.g. 6 years for a 4-year degree) reported 
by our partner institutions was 29 percent. Of the 20 institutions that participated in N2FL, three 
are community colleges and three are four-year colleges in the City University of New York 
system; seven are community colleges in the Virginia Community College System; three are 
community colleges in Texas; two are four-year public universities in University of Texas system; 


and two are four-year public institutions in Ohio and Washington State. 


3.1. Eligibility Criteria and Sample 
Degree-seeking students were eligible to participate in the study if they: 1) were actively 
enrolled, 2) had an active cell phone number on record with their institution, and 3) completed at 


least 50 percent of the credits typically required for degree completion prior to intervention 


those sessions were not knowing what steps to take to graduate and where to turn for help on campus when challenges 
arose. 


launch.’ Although the goal of the intervention was to provide enhanced support to students at risk 
of late dropout, we established broad eligibility criteria to examine heterogeneity in treatment 
effects by predicted risk of dropout. 

Based on the eligibility criteria above and the size of enrollments at our partner institutions, 
we recruited 21,533 students to participate in the study. Of this experimental sample, we randomly 
assigned 13,826 to the treatment group and 7,727 to the control group. Students assigned to the 
control condition did not receive any text messages as part of the intervention but maintained 
access to the support structures typically available on their campus. However, as discussed above, 
outreach to students, especially upper-division students, is limited at many public colleges and 
universities. Therefore, the relevant counterfactual is that control group students did not receive 
personalized support unless they had the time, motivation, and awareness to seek it out. 

In columns 2-3 of Table 1, we present summary statistics by treatment status for the 
students in the analytic sample. To examine the extent to which the sample reflects the population 
of undergraduates attending broad-access, public institutions nationally, we report (in column 1) 
analogous statistics for a nationally representative sample using data from the National 
Postsecondary Student Aid Study of 2012 (NPSAS:12). Finally, we report (in column 4) balance 
between the treatment and control experimental conditions. 

Across both treatment and control groups, approximately 43 percent of students in the 
study sample were male, 50 percent were students of color, and the average age of students at the 
start of the intervention was 21.5 years. Approximately half of our experimental sample attended 


two-year institutions and half attended four-year institutions. Students had earned an average of 


7 At two-year institutions, students in pursuit of associate degrees who had completed 30 or more college-level credits 
were eligible to participate. At four-year institutions, bachelor’s degree-seeking students who had completed 60 or 
more college-level credits were eligible for the study. 


65 college-level credits and completed 91 percent of the credits they had attempted prior to the 
start of the intervention. Students in the study sample on average had a 30 percent chance of 
dropping out prior to earning their degree based on the predictive models we developed using 
historical data from partner institutions (see section 4.3 for more details on these models). 

Our experimental sample is fairly representative of the national student population 
attending broad-access public institutions with respect to sex (43 percent male versus 44 percent) 
and racial/ethnic composition (50 percent in both samples). Based on our institutional recruitment 
strategy, students attending four-year institutions are overrepresented in our sample (48 percent 
versus 22 percent nationally). As a result, on average the students in our study are slightly younger 


than the typical enrollee at public broad-access institutions (21.6 years versus 27.1 years). 


3.2. Intervention Design 

N2FL consisted of a pilot phase (2016-2017 academic year) and a subsequent scale phase 
(2017-2018 and 2018-2019 academic years). Across both phases, 21 institutions participated in 
the study. Nine institutions participated in the pilot phase, eight of which also participated in the 
scale phase. We recruited an additional 12 institutions, for a total of 20, to participate during the 
scale phase. With the exception of one institution that only participated in the pilot phase, we 
estimate intervention impacts off a sample that includes participants in the pilot phase, scale phase, 
or both.® All nine pilot institutions used a text messaging model and platform called Persistence 


Plus, whereby both the automated messages and follow-up responses (to students who wrote back) 


8 We exclude 500 students at one institution that only participated in the pilot phase because we observe a large initial 
enrollment difference between treated and control students at that campus. During the pilot phase, we randomized 
students in late summer before fall enrollments finalized and message outreach began after classes started. The 
imbalance therefore occurred due to the timing of randomization, not as a result of message outreach, and would likely 
bias estimates of intervention impacts. 
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were automated and personalized to students’ use of keywords in their response.’ Two of the pilot 
institutions, Ohio University and University of Washington-Tacoma, continued to use Persistence 
Plus during the scale phase. 

The other eighteen scale phase institutions adopted an interactive two-way text messaging 
campaign that actively promoted opportunities for students to connect with advisors at their 
campus directly via text. Eligible students who were randomly assigned to treatment received 
approximately one pre-scheduled text message each week over the course of 2-3 semesters, 
depending on the institutional partner. These messages were sent automatically by the text 
messaging vendor Signal Vine according to a predetermined content schedule and delivery 
timeline that we developed collaboratively with advisors at each partner institution. We provide a 
sample of message content in Appendix 1. 

We present in Table 2 the start and end dates of messaging, the number of terms over which 
students were messaged, and student and advisor engagement statistics for each of our twenty scale 
phase institutions. Messaging start and end dates depended on each institution's preference. Most 
institutions began messaging students during the 2018 calendar year. Students at most institutions 
received automated messages for 2-3 semesters, depending on each institution’s preference. !° 

The topics and frequency of scheduled messages stayed fairly consistent across institutions. 
Messages were sent approximately once per week and prompted students to complete important 
tasks (e.g., register for the next semester’s courses), encouraged them to use campus resources 


(e.g., tutoring centers, financial aid office), and addressed feelings of stress and anxiety (e.g., 


° For example, during the spring term students who reported uncertainty about their remaining math requirements 
received the following messaging: “Last semester you were unsure whether you had any math requirements left to 
graduate. Were you able to get that sorted out?”. Students who replied “Yes” then received the following response: 
“Fantastic! If you’re currently taking any math courses remember that you can always visit the Math Lab in [on- 
campus location] for free tutoring.” 

'0 Pilot phase institutions had six semesters of messaging: two semesters during the pilot phase and four semesters 
during the scale phase. 
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financial hardships). We worked with each partner institution to tailor pre-scheduled message 
content to their institutional context, such as inserting the name of campus-specific tutoring centers 
or modifying the tone to fit their student population. 
The messages leveraged several key behavioral insights: (1) Increase informational salience: 
To simplify the process of accessing on-campus resources, one set of messages encouraged 
students to connect with campus-based academic and financial resources and provided them with 
specific contact and location information where assistance was available.'! (2) Promote 
implementation intentions: A second set of messages reminded students of upcoming deadlines 
and encouraged them to make implementation plans that increase the likelihood of task completion 
(Milkman, Beshears, Choi, Laibson, & Madrian, 2011; Nickerson & Rogers, 2010).'* (3) Set 
positive social norms: A third set of messages amplified descriptive informational norms to 
motivate action (Cialdini et al, 2016; McDonald & Crandall, 2015).!° 
One distinguishing feature of the N2FL intervention was the ability for students to write 
back to the scheduled messages with any questions or requests for help and get connected with 
campus advisors via text. Indeed, most scheduled messages encouraged students to text back by 
posing a final question designed to encourage student response and engagement. Each partner 
institution identified a specific advisor or staff team to monitor the text messaging inbox for student 


replies and to respond to students’ questions or requests for assistance. 


‘| For example, the following message encouraged students to use tutoring resources: “Hi [student name], using the 
[name of campus tutoring center] can help you do well on midterms & boost your grades. Can I help connect you?”. 
2 For example: “Hi [student name], Summer and Fall 2019 registration opens today. Don’t miss your chance to secure 
a seat in the courses you need to graduate. What day do you plan to register?” 

'3 For example: “Hi! Did you know 580,000+ New Yorkers filed FAFSA by this day last year? Join your peers and 
visit FAFSA.gov now to get the most aid.” Some messages moreover embedded infographics to reinforce the call to 
action and increase the salience of relevant information. 
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Staffing models varied across institutions: Some institutions elected to use professional or 
faculty advisors, while others appointed general staff (e.g., administrative assistants) to staff the 
messaging inbox and reply to students who texted in. The language of scheduled messages were 
modified to match the nature and scope of each designated staff’s role. Specifically, institutions 
whose professional advisors had the capacity to support students directly (e.g., choosing which 
courses to register for or filling out the FAFSA) sent automated messages that offered direct 
assistance. In contrast, institutions that used a general administrative assistant offered assistance 
with connecting with the appropriate resources. A summary of the four primary staffing models 
that emerged can be found in Appendix Table Al. 

As we also show in Table 2, message engagement rates varied substantially among both 
students and advisors across institutions. Student response rates were generally high across 
institutions, with approximately half or more of students responding at all institutions. That being 
said, institutional-level student response rates ranged from a low of 44 percent at Blinn College 
(Texas) to a high of 78 percent at Lehman College (CUNY). The average institution-level student 
response rate was 58 percent. Advisor response rates (to messages sent by students) also tended to 
be quite high, though there was more heterogeneity in institution-level advisor response rates than 
for student response rates: advisor response rates ranged from as low as 33 percent at 
Kingsborough Community College (CUNY) to a high of 88 percent at Thomas Nelson Community 
College (VCCS). Because higher or lower advisor response rates affected a key design component 
of the intervention (the availability of advising via text), we investigate whether this feature of 


institutional heterogeneity was correlated with N2FL efficacy. 


3.3. Data and Measures 


13 


The data for this study consists of student-level administrative records maintained and 
provided by our institutional partners for both study participants and previous cohorts of students. 
The specific data elements vary across schools due to availability, but in general we observe 
baseline demographic and academic measures (e.g., gender, race, high school GPA and college 
entrance exams, etc.) and term-by-term records of students’ financial aid receipt, enrollment 
intensity (e.g., credits attempted), academic performance (e.g., credits completed, term and 
cumulative GPA, etc.), and degree receipt. Most of our partner institutions also routinely collect 
enrollment and degree information from the National Student Clearinghouse (NSC) on previously 
enrolled students. We also relied on NSC data when it was available to capture transfer, enrollment, 
and degree information at non-participating institutions. 

We use these data in three ways. First, we used the historical data provided by each 
institution to develop school-specific dropout prediction models. We present details about the 
model construction process in Appendix 2 and report descriptive statistics of the study sample by 
tercile of predicted dropout risk in Appendix Table A2. Second, we use the data to assess whether 
students randomly assigned to the treatment and control conditions appear to be equivalent in 
expectation on observable and unobservable dimensions. Third, we use the data to evaluate the 
impact of the intervention on students’ academic progress spanning different time horizons. In our 
main tables we focus on impacts within four terms of the start of the intervention--the longest time 
horizon we can observe for all institutions.'* In appendices we report impacts of the intervention 
for the subset of earlier-participating interventions for which we can observe outcomes six terms 
after the start of the intervention. We report on four primary outcome measures over these time 


horizons: whether students re-enrolled or graduated, the cumulative number of credits earned 


'4 One of the institutions only provided data through three terms following the intervention. To preserve our sample 
we include this institution’s students in our main tables. Results are robust to excluding this institution as well. 
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following intervention, whether students graduated, and for students attending community 


colleges, whether they transferred to a four-year institution. 


3.4. Randomization Procedure and Baseline Equivalence 

To investigate whether impacts of message outreach varied with predicted risk of dropout, 
we randomly assigned students to receive message outreach using a block randomization 
procedure that afforded greater statistical power to examine evidence for heterogeneity of 
treatment effects.'> We implemented this procedure by predicting the probability of dropout for 
currently enrolled students using the dropout models we developed (described in Appendix 2). 
Within each institution, we then ranked students by dropout risk and randomly assigned students 
with similar probabilities of dropout to either the treatment or control conditions.'® 

In column 4 of Table 1, we show that random assignment appears to have created 
equivalent groups of students in the treatment and control conditions. In both tests of equivalence 
on individual covariates or in our test for joint equivalence across all covariates, we fail to detect 
any significant differences between treatment and control students. And as we show in Appendix 
Tables A3-A5, the treatment and control groups are also well-balanced within each of our three 


higher education system partners (CUNY, THECB, and VCCS). 


3.5. Empirical Strategy 


'S Tn our study proposal to the Institute of Education Sciences, we proposed to examine heterogeneity on this student 
background dimension alone. We therefore designed our study with this analysis in mind. 

'6 At most institutions students we randomly assigned to one of three treatment arms: a control condition and two 
variants of the treatment group, one of which received a set of messages focused more on academic barriers and 
another which received a set of messages tailored more to address financial obstacles (though students in both groups 
received messages about both academic and financial barriers and resources). However, in all analyses we aggregate 
treated students into a pooled treatment group because we do not observe evidence of differential effects by variant of 
message outreach. 
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To evaluate the effects of message outreach on academic progress and performance, we 
estimate intent-to-treat (ITT) models of the following form using ordinary least squares or linear 
probability models: 

(1) Yip = a + BTiy + Oy + CXin + Ein, 

where Y;, is one of the four academic outcomes described above for student i in randomization 
block b. Tj, is the treatment indicator set to one for students assigned to receive text-message 
support and zero otherwise. 56, denotes randomization block fixed effects. The coefficient of 
interest in this model is 8, which represents the causal estimate of being assigned to receive text- 
based outreach. The set of student-level covariates (X;,) is comprised of indicators for sex, 
race/ethnicity (Black, Hispanic, Other, and Missing Race), number of credits earned prior to the 
intervention, percent of attempted credits earned prior to the intervention, and whether the student 
had ever transferred prior to the intervention. We do not include campus fixed effects in the model, 
as time-invariant differences across campuses are already controlled for through the block 
dummies. €;, is a student-specific random error term, and in all results we report robust standard 
errors that allow for heteroskedasticity in the error term. 

We examine heterogeneity of treatment effects by dropout risk by estimating models of the 
following form: 

(2) Yiok = Liar Be (Ti * DR) + 5p + 5%" + Ey, 

where, as before, i and b respectively index students and blocks, and DR;,, is an indicator for 
whether a student’s predicted probability of dropout is in tercile k. All other terms in the model are 
defined as above. This specification allows for estimation of treatment effects separately by tercile 
of predicted dropout risk, whereby tercile one categorizes students with low relative risk of 


dropout, tercile two captures students with medium risk of dropout, and tercile three denotes 
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students with high predicted risk of dropout according to the college-specific dropout prediction 


models we developed using historical data from each institution. 


4. Results 


4.1 Overall impacts 


In the top panel of Table 3, we present estimates of N2FL’s impact on our primary 
outcomes of interest, measured four terms following the start of the intervention at each institution. 
Across experimental conditions, most students (76.2 percent) re-enrolled or graduated within four 
terms, and the N2FL interactive text messages did not significantly increase re-enrollment or 
graduation rates. We can rule out impacts of 1.8 percentage points or larger on the probability of 
re-enrollment or graduation. We similarly do not observe impacts of the treatment on the number 
of credits students accumulated. Likewise, when we investigate impacts of N2FL on degree 
attainment alone, we also find no significant effects. Fifty-nine percent of the control group 
completed their degree within four terms, and we can rule out treatment impacts of 0.9 percentage 
points or larger. Among students at two-year institutions, we find no impact of N2FL on transfer 
to four-year institutions and we can rule out impacts of 1.0 percentage points or larger. Finally, in 
the bottom panel of Table 3, we show that there are similarly no effects of N2FL within six terms 
of the intervention for the subset of institutions for whom we can observe outcomes over that time 
frame. None of our estimates is significant, and if we were to apply multiplicity adjustments given 
the number of estimated impacts (here and throughout the paper), it would only further accentuate 


our lack of identification of significant impacts. 


4.2 Impacts by predicted baseline risk of withdrawal 
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In the top panel of Table 4, we present heterogeneous impacts of N2FL on the same 
outcomes measured four terms following the start of the intervention by tercile of predicted risk 
of withdrawal. As expected, we observe the highest rates of re-enrollment or completion, credit 
accumulation, degree attainment, and transfer (among two-year enrollees) among students in the 
bottom tercile of risk. For instance, 70.7 percent of students in the control group in the bottom 
tercile earned a degree within four terms, compared with 45.9 percent of control students in the 
top tercile. Once again, we do not observe significant impacts of N2FL across any of the risk 
terciles on any of the primary outcomes, and in all cases can rule out even moderate treatment 
effects. In the bottom panel of Table 4, we show that the null effects of N2FL across the distribution 
of predicted risk holds whether we measure impacts six terms following the start of the 


intervention. 
4.3 Impacts by higher education system 


We implemented N2FL across five states, with most institutions a part of three higher 
education systems: the City University of New York (CUNY), the Texas Higher Education 
Coordinating Board (THECB), and the Virginia Community College System (VCCS). These 
systems differ in their governance structure, public expenditures in higher education, institutional 
context, and student composition, so it is possible the impacts of N2FL would vary across systems. 
We investigate whether this is the case in Table 5. As with the absence of heterogeneity by 
predicted risk, we find no evidence of impacts of N2FL four terms post-intervention across any of 
the three higher education systems, and we can rule out the possibility of moderately-sized effects. 
We similarly do not find significant impacts by system six terms post-intervention, as we show in 


Appendix Table A6. We show in Appendix Table A7 that the impacts of N2FL moreover do not 
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vary by predicted baseline risk, within each higher education system. Finally, as we show in 


Appendix Table A8, the impacts of N2FL do not vary by two- or four-year institution. 
4.4 Mechanisms 


While we maintained a consistent core of scheduled message content across institutional 
partners, by virtue of working with 20 institutions, there were still potentially important differences 
in treatment implementation and institutional context that could lead to differences in N2FL 
efficacy. For instance, as we describe above, institutions varied in whether the staff member 
responding to student messages was a dedicated professional advisor, part of a team of advisors, 
or a non-advisor staff member who made connections to other advisors on campus. Institutions 
also varied in their overall level of advising support (which we proxy for using the advisor:student 
ratio), whether institutions required students to meet with academic advisors, and whether they 
had other texting campaigns operating in parallel with N2FL. 

In Appendix Table A9, we investigate whether treatment efficacy varied based on any of 
these factors. We treat these investigations as exploratory since we are underpowered to detect 
impacts across numerous sub-groups. Nonetheless, we fail to find any significant differences in 
N2FL efficacy by whether institutions used professional advisors, a team of advisors, a staff 
“connector”, or in the case of two institutions, automated responses (Appendix Table A9, Panel 
A). Nor do we find significant differences by whether institutions had larger or smaller 
advisor:student caseloads, required students to meet with advisors, or had parallel texting 
campaigns in place (Appendix Table A9, Panel B). 

Finally, as we describe above, we observed meaningful heterogeneity in advisor response 
rates across campuses. In Appendix Figure Al, we investigate whether the impacts of N2FL varied 


by advisor responsiveness to text messages students sent in response to scheduled outreach they 
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received. Appendix Figure A1 plots treatment effects on our primary outcomes four terms post- 
intervention by quartile of advisor responsiveness.'’ We find no evidence of N2FL impacts on any 
of the main outcomes across the distribution of advisor responsiveness; furthermore, the 
confidence intervals of the effect estimates by quartile of advisor responsiveness overlap 
considerably. We conclude that N2FL had no impact on the likelihood of college persistence or 


degree completion, even among students paired with highly responsive and engaged advisors. 


5. Discussion 

Many college students within reach of graduation remain at risk of dropping out before 
they earn a degree. Although leaving without a degree may be a rational human capital 
investment decision for some, reducing late dropout is likely to benefit many near completers 
given the prevalence of the phenomenon and the high returns to degree completion for most 
college enrollees. We developed the N2FL intervention to examine if text-based outreach offers 
a scalable solution to support students at risk of late dropout while they remain enrolled in 
college. To our knowledge, previous interventions targeted to this population have strictly 
attempted to re-engage individuals after they have already withdrawn from school. The findings 
in this study provide strong evidence that low-touch interventions such as text-based outreach 
may not be an effective policy tool to reduce the incidence of late dropout from college. We 
estimate null impacts on persistence and completion in the overall sample, separately by 
students’ baseline predicted risk of dropout, and across numerous dimensions of treatment 


fidelity and institutional context. 


'7 The within-quartile outcome means for Panel A are: Q1 = 75.6%; Q2 = 65.8%; Q3 = 75.1%; and Q4 = 74.2%. The 
within-quartile outcome means for Panel B are: Q1 = 29.9; Q2 = 29.4; Q3 = 31.0; and Q4 = 24.2. The within-quartile 
outcome means for Panel C are: Q1 = 58.5%; Q2 = 48.4%; Q3 = 60%; and Q4 = 42.0%. 
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The most immediate question is what explains the null impacts of N2FL. Our findings are 
consistent with several other recent nudge interventions in the education arena that have not 
scaled successfully (Bergman, Denning, and Manoli 2019; Bird et al. 2021; Gurantz et al. 2021). 
However, unlike N2FL, those interventions relied on messages delivered from organizations 
with which students did not have close, pre-existing connections, leaving open the possibility 
that the efficacy of outreach campaigns requires participation of local entities that students trust. 
We find null impacts in this study despite partnering with institutions to design message content 
and sequencing, sending all messages from a specific advisor at the students’ institution, and 
encouraging students and advisors to interact in real-time using two-way texting capabilities. The 
findings in this study therefore suggest that nudging at scale in postsecondary education is often 


not effective for other reasons. 


We posit three alternative explanations for the null findings in this study. One possibility 
is that the messages were not salient enough to students to foster meaningful engagement with 
advisors on campus. Text-based outreach has become increasingly widespread over the past 
decade and colleges must compete more in recent years for the attention of students. Although 
we observed high student and advisor response rates in N2FL overall, we cannot rule out that 
college students may have reached a point of text message saturation, such that the efficacy of 


outreach campaigns launched five or 10 years ago will be more limited today. 


Alternatively, because N2FL relied on the existing advising infrastructure of colleges and 
universities to engage with students, it is possible that the intervention asked too much of college 
staff with large caseloads and competing demands. This may be especially true in the context of 
upper-division students at risk of dropout, who may face acute academic and financial barriers 


that require more intensive assistance than two-way texting or traditional models of advising can 
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provide (Mabel & Britton, 2018; Ortagus, Skinner & Tanner, 2020). The intensity of support at- 
risk students need may also explain the success of more resource-intensive interventions in 
college like one-on-one coaching programs (Bettinger & Baker, 2014; Oreopoulos & 
Petronijevic, 2018), which often have low student-coach caseloads and augment, rather than 
depend on, the traditional advising capacity of colleges. A third possibility is that N2FL may 
have engaged students too late into their college careers. As evident from the promising impacts 
of comprehensive college support interventions (Dawson, Kearney & Sullivan, 2021; Evans et 
al., 2020; Weiss et al., 2019), there may be important benefits to programs that engage students 
throughout their college career. If that is the case, then upper-division students at risk of dropout 


may benefit most from interventions that begin earlier and offer continuous support. 


Although we are unable to pin down the precise reason(s) why N2FL produced null 
impacts, our findings are clear that college students at risk of late dropout likely require higher- 
touch intervention than can be delivered via text-based outreach. Yet the reality is that high- 
touch interventions are expensive and many colleges, especially broad-access institutions that 
serve most students at risk of late dropout, operate on tight budgets. Helping more college 
students cross the finish line will require institutions to target resources to at-risk students who 
stand to benefit most. We believe predictive analytics has an important role to play in this regard. 
We embedded predictive modeling into the design of N2FL to help colleges identify which 
students experienced the largest gains from message outreach. While we find no impacts of 
message outreach on persistence and completion across the distribution of predicted baseline 
risk, the null effects found in this study do not reflect that predictive models convey limited 
utility for colleges. To the contrary, coupling predictive analytics with ineffective interventions 


conceals the potential for using the former to increase postsecondary attainment. 
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Further research is needed to determine if more intensive student support interventions 
that have proven effective in other contexts can lower rates of late dropout from college. To 
maximize the cost-effectiveness of those strategies, we encourage researchers to embed 
predictive analytics into future research, as we have done in this study, to help policymakers and 
college leaders better distinguish between marginal and inframarginal students. We believe this 
is the most feasible strategy for reconciling the tension between the resource-intensive supports 
that many college students appear to need and the resource-constrained environments in which 


most higher education institutions operate. 
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Table 1. Pre-treatment characteristics of experimental sample by treatment condition and summary 
statistics of nationally representative sample of undergraduates attending public institutions 


#** H<0.01 ** p<0.05 * p<0.10 


(1) (2) (3) (4) 
Experimental Sample 
NPSAS Treated Control T-C 
Sample Students Students Difference 

Male 0.438 0.433 0.442 0.000 
Black 0.175 0.153 0.146 -0.001 
Hispanic 0.220 0.223 0.208 -0.003 
White 0.507 0.388 0.416 0.003 
Race other 0.098 0.129 0.130 0.001 
Race missing 0.000 0.108 0.100 -0.001 
Age 27.10 21.58 21.39 0.102 
Enrolled in public 2-year institutions 0.781 0.522 0.467 0.000 
Enrolled in public 4-year institution 0.219 0.478 0.533 0.000 
Cumulative credits earned before 

intervention 61.76 65.98 -0.205 
Share of credits earned before intervention 0.906 0.908 0.000 
Transferred into current school 0.295 0.308 -0.006 
Predicted risk of dropout 0.297 0.294 0.000 
P-value on F-test for joint significance 0.741 
Number of Students: 58,410 13,826 Cee 21555 


Notes: The data in column | is from the National Postsecondary Student Aid Study of 2012 (NPSAS:12) and is 
restricted to students attending public, urban two- and non-selective four-year institutions. Summary statistics in 
column | are calculated using survey sampling weights. The data in columns 2-4 are from partner institution 
administrative records. Means are reported in columns 2 and 3. Estimates of post-randomization balance are 
reported in column 4 from OLS/LPM models that include randomization block fixed effects. 
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Table 2. Engagement statistics for students and advisors at partner institutions, presented by system 


Panel A: VCCS Institutions 


(1) (2) (3) (4) (5) (6) (7) 
Memau Piedmont Cone John Tyler Wytheville Themes Germanna 
Empire Virginia Virginia ; : Nelson : 
: : : Community Community . Community 
Community Community Community College College Communit Collese 
College College College é & y College 8 
Start of messaging Fall ‘18 Fall ‘16 Fall ‘18 Spring ‘18 Fall ‘18 Fall ‘16 Spring ‘18 
End of messaging Fall ‘19 Spring ‘19 Fall ‘19 Fall ‘19 Fall ‘19 Spring “19 Spring ‘19 
Terms messaged 3 6 3 4 3 6 e) 
Percent students responded 44.74 50.49 50.59 51.31 57.14 53.77 59.46 
Percent students opt-out 65.79 29:15 26.47 29.64 54.76 19.82 31.42 
Advisor response rate 48.21 88.03 83.55 67.13 36.43 88.43 64.8 
N Treated Students 76 629 171 766 84 867 296 
Panel B: CUNY Institutions 
New York , : 
City College LaGuardia Kingsborough Sohn lay iene School of 
Community | Community Professional 
of College College : 
College College Studies 

Technology 
Start of messaging Spring ‘18 Fall ‘16 Spring ‘18 Spring ‘18 Fall ‘16 Fall ‘16 
End of messaging Spring ‘19 Spring “19 Fall ‘18 Spring “19 Fall ‘19 Spring ‘19 
Terms messaged 3 6 2 3 6 6 
Percent students responded 50.05 58.34 61 62.41 78.09 76.06 
Percent students opt-out 7.34 13.93 13 33:54 16.01 17.57 
Advisor response rate 30.97 50.31 33.11 pees | 87.58 23.2 
N Treated Students 756 1,333 622 1,368 751 389 
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Table 2, continued. Engagement statistics for students and advisors at partner institutions, presented by system 


Panel C: Texas Institutions 


Austin University of ike University of 
Blinn College Community Texas at Colleges Texas of the 
College Arlington Permian Basin 
Start of messaging Fall ‘18 Fall ‘18 Fall ‘18 Spring ‘19 Fall ‘16 
End of messaging Spring ‘19 Fall ‘19 Fall ‘19 Fall ‘19 Spring ‘19 
Terms messaged 2 3 3 2 a 
Percent students responded 44.03 47.70 56 62.07 68.80 
Percent students opt-out 8.85 60.97 30.1 17.73 28.11 
Advisor response rate 77.64 71.34 77.34 T23 61.19 
N Treated Students 1,000 423 1,000 608 659 
Panel D: Other Institutions 
; niversity of 
Ole : ae 
University 
Tacoma 
Start of messaging Fall ‘16 Fall ‘16 
End of messaging Spring '19 Spring '19 
Terms messaged 6 6 
Percent students responded 61.04 61.59 
Percent students opt-out 40.97 28.12 
Advisor response rate -- -- 
N Treated Students 1,003 1,025 
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Table 2, continued. Engagement statistics for students and advisors at partner institutions, presented by system 


Panel E: All Institutions 


All Institutions 
Percent students responded 58.36 
Percent students opt-out 24.78 
Advisor response rate 66.16 
N Treated Students 13,826 


Notes: This table presents student and advisor engagement statistics for the institutions included in the study. Note that some 
institutions had multiple cohorts of students; in that case, we averaged the student response rate and student opt-out rate 
based on cohort sample size. Note that advisor response rate is averaged across each cohort and we do not have data on 
advisor response rate from the Fall 2016 cohorts. The start date presented for each institution is that of the earliest cohort at 
that institution. Some institutions performed bulk opt-outs for students; those institutions are: all VCCS institutions; the 
School of Professional Studies; John Jay College; Lehman College; Austin Community College; the University of Texas at 
Arlington; and Alamo Colleges. Finally, advisor response rate is inapplicable for Ohio University and the University of 
Washington Tacoma since responses were automated at those institutions. 
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Table 3. Estimates of intervention effects on academic outcomes 


#4 H<0.01 ** p<0.05 * p<0.10 


(1) (2) (3) (4) 
Re-Enrolled or Number py Earned Transferred to 
Graduated Credits Degree Four-Year 
ae Accumulated 8 
Four Term Outcomes: 
Treatment Impact 0059 421 -.0027 -.008 
(0.006) (0.263) (0.006) (0.009) 
Control Mean .762 34.823 591 459 
Observations 21553 21553 21553 10534 
Six Term Outcomes: 
Treatment Impact -.0036 088 -.0065 -.011 
(0.007) (0.416) (0.007) (0.012) 
Control Mean 815 42.623 755 526 
Observations 12879 12879 12879 6788 


Notes: Estimates are from OLS/LPM models that include randomization block fixed effects, and 
the following pre- treatment covariates: indicators for sex, race/ethnicity (Black, Hispanic, 
Other, and Missing Race), and transfer status at the start fall 2016, as well as continuous 
measures of cumulative credits completed, and the fraction of total credits attempted that were 
earned at the start of the intervention. Robust standard errors are reported in parentheses. 


Column 4 only includes students at 2-year colleges. 
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Table 4. Estimates of intervention effects by tercile of dropout risk and outcome horizon 


(1) (2) (3) (4) 
Re-Enrolled or Numer oF Earned Transferred 
Graduated reais Degree to Four-Year 
Accumulated 
Four Term Outcomes: 
Treatment x Bottom Tercile 01 .614 005 -.022 
(0.008) (0.440) (0.010) (0.017) 
Treatment x Middle Tercile 0 314 -.004 005 
(0.010) (0.448) (0.011) (0.016) 
Treatment x Top Tercile .0076 329 -.0093 -.0069 
(0.011) (0.479) (0.011) (0.015) 
P-value on F-test of Equal 
Effects athe .867 .603 522 
Control Mean - Bottom Tercile 853 S20 .107 .602 
Control Mean - Middle Tercile .785 35.461 .607 453 
Control Mean - Top Tercile .649 31.489 459 B20 
Observations 21553 21553 21553 10534 
Six Term Outcomes: 
Treatment x Bottom Tercile -.009 337 -.003 -.017 
(0.009) (0.707) (0.011) (0.020) 
Treatment x Middle Tercile 002 231 -.007 -.001 
(0.011) (0.707) (0.012) (0.021) 
Treatment x Top Tercile -.0041 -.295 -.0091 -.014 
(0.014) (0.753) (0.014) (0.020) 
P-value on F-test of Equal 
Effects tot 81 939 852 
Control Mean - Bottom Tercile a 45.4 867 651 
Control Mean - Middle Tercile 834 42.955 .776 2 
Control Mean - Top Tercile 701 39.51% 621 401 
Observations 12879 12879 12879 6788 


*#E Y<().01 ** p<0.05 * p<0.10 

Notes: Estimates are from OLS/LPM models that include risk rating, randomization block fixed effects, 
and the following pre- treatment covariates: indicators for sex, race/ethnicity (Black, Hispanic, Other, 
and Missing Race), and transfer status at the start fall 2016, as well as continuous measures of 
cumulative credits completed, and the fraction of total credits attempted that were earned at the start of 
the intervention. Robust standard errors are reported in parentheses. Risk ratings terciles are defined 
within an institution. Column 4 only includes students at 2-year colleges. 


29 


Table 5. Estimates of intervention effects on academic outcomes by system 


##* H<0.01 ** p<0.05 * p<0.10 


(1) (2) (3) (4) 
Re-Enrolled or Number of Transferred to 
Credits Earned Degree, 
Graduated, Four-Year, 
Accumulated, Four Terms 
Four Terms Four Terms 
Four Terms after 
after ; after 
. after Intervention : 
Intervention : Intervention 
Intervention 
Panel A: VCCS Institutions 
Treatment 
Impact -.005 635 -.013 .004 
(0.014) (0.457) (0.015) (0.014) 
Control Mean .672 22.847 77 325 
Observations 4336 4336 4336 4336 
Panel B: CUNY Institutions 
Treatment 
Impact .002 -.453 -.008 -.02 
(0.009) (0.410) (0.011) (0.015) 
Control Mean 785 32.098 633 597 
Observations 7751 7751 7751 4064 
Panel C: TX Institutions 
Treatment 
Impact 0.011 0.969* 0.005 
(0.012) (0.533) (0.011) 
Control Mean 0.704 25.65 0.446 
Observations 5535 5535 5535 


Estimates are from OLS/LPM models that include risk rating, randomization block fixed effects, 
and the following pre- treatment covariates: indicators for sex, race/ethnicity (Black, Hispanic, 
Other, and Missing Race), and transfer status at the start fall 2016, as well as continuous 
measures of cumulative credits completed, and the fraction of total credits attempted that were 


eared at the start of the intervention. Robust standard errors are reported in parentheses. Column 


4 only includes students at 2-year colleges. 
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Appendix 1: Sample Text Message Content 


March 12, 
2018 3:00 PM 


March 14, 
2018 3:00 PM 


March 20, 
2018 3:00 PM 


March 26, 
2018 3:00 PM 


March 26, 
2018 3:00 PM 


March 28, 
2018 3:00 PM 


March 28, 
2018 3:00 PM 


April 4, 2018 
3:00 PM 


April 4, 2018 
3:00 PM 


Spring 2018 Semester 


Hi [first_ name], I’m <advisor name>, a <institution> advisor, & I’m here to 
support you to finish your degree! Is there anything I can help you with 
now? 


Hi [first name]! Early registration for fall term starts 3/26. Can I help you 
choose courses that will help you finish & earn your degree? 


Hi, it’s <advisor name>. Students share they miss meals & face other 
financial hardships. We have resources for those facing these challenges. 
Can I help connect you? 


Hi, it’s <advisor name>. FAFSA.gov is open for the 2018-2019 school year, 
and applying early gets you the most financial aid. Have you started FAFSA 
yet? 


Hi, it’s <advisor name>. Lots of students get grants that help them finish 
their degree and graduate to a rewarding career. Can I help you do FAFSA 
this year? 


<Infographic with average financial aid award for students at this 
institution> 


Hi [first_name]! Putting in time at the <campus tutoring center name> math 
and writing centers will help you succeed in your classes! Can I help you get 
connected there? 


<infographic with information about the campus tutoring center> 


Hi [first_ name]! Doing well in classes can bring you closer to graduating & 
a higher income! Can I help connect you to the <campus tutoring center 
name> math and writing centers? 


<infographic with information about the campus tutoring center> 


Hi [first_ name], I want you to graduate! Some students take courses that 
don’t count for their degree. Can I help you register for classes that fit your 
program? 


<infographic with visualization of time to degree for students at institution 
who take classes that count toward their degree vs. those that do not> 


Hi [first_name], I know college can feel expensive. Can I help you register 
for classes that fit your program, so you don’t pay for courses you don’t 
need? 
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oa <infographic with visualization of relationship between excess degree t> 


April 10, 2018 


April 10, 2018 


April 17, 2018 


April 17, 2018 


April 23, 2018 


April 25, 2018 


April 25, 2018 
3:00 PM 


May 8, 2018 
3:00 PM 


August 22, 
2018 4:00 PM 


August 27, 
2018 4:00 PM 


Hi, [first_ name]! Now’s a great time to think about summer. Taking courses 
can bring your diploma much sooner. Want to look into summer courses 
together? 


Hi, [first_ name]! Now’s a great time to think about summer. Taking courses 
can bring your diploma and a good-paying job much sooner. Can I help you 
look into courses? 


Hi! Have you had a chance to apply for financial aid for next year? 
FAFSA. gov is now open. Applying early gets you the most aid. Let me 
know if I can help. 


Hi! Did you know 268,000+ Virginians filed FAFSA by this date last year? 
Join your peers and visit FAFSA.gov now to get the most aid. Can I help? 


Hey, it’s <advisor name>. With finals coming up, I wanted to check if 
you’ve used SMARThinking online tutoring or the <campus tutoring center 
name>. Can I help you get connected? 


<infographic with visualization of relationship between using academic 
supports and student academic performance> 


<R: TNCC_academic_performance> 


Doing well on exams brings you closer to your degree. Pick 2 hours each 
day to study using this calendar. Can I help you make a study schedule? 


<R: TNCC_study_calendar> 


Studying for finals pays off! Grads in [program] earn an average of 
$[earnings] per year! Can I help make a study schedule? 


Hey, it’s <advisor name>. Congrats on finishing another semester--you’re 
one term closer to your degree. Ill reach out again in the fall. Happy 
summer! 


Fall 2018 Semester 


Hi [first name], it’s <advisor name>. I'm happy to have you back on 
campus! I’m here to help you continue staying on track. How do you feel 
about the start of the term? 


<R: tnec_f18_introtips> 


Hi! 500+ <institution> students used resources like the <campus tutoring 
center name> last spring to pass classes & move toward graduation. Do 
you have the academic support to succeed? 
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September 5, 
2018 4:00 PM 


September 17, 
2018 4:00 PM 


September 24, 
2018 4:00 PM 


October 1, 
2018 4:00 PM 


October 1, 
2018 4:00 PM 


October 8, 
2018 4:00 PM 


October 15, 
2018 4:00 PM 


October 15, 
2018 4:00 PM 


October 22, 
2018 4:00 PM 


<R: tnec_f18_mathcenter> 


Hey, it’s <advisor name>. The start of the fall semester can be 
challenging for students who juggle classes w/ family and work. How’s 
the transition going for you? 


Hi [first_ name], you may be on track to graduate soon--congrats! Last day 
to apply for Fall 2018 graduation is 10/1. Can I help you do a graduation 
check? 


<R: tnec_f18_gradapply> 
Hi [first_name]! Priority registration for Spring semester starts 10/22. Can 


I help you check which courses you still need to graduate? 


Hi! FAFSA. gov opened this week for the 2019-20 school year. Apply 
early to get more $$ toward your courses. Can I help you apply? 


<R: tnec_f18_ fafsa_1> 


Hi! FAFSA. gov opened this week for the 2019-20 school year. Apply 
early to get all the free $$ you’re eligible for. Can I help? 


<R: tnec_f18_fafsa_1> 


Hi. I know financial stress like budgeting or loans can distract students 
from focusing on school & graduating on time. Do you face these 
challenges? 


<R: tnec_f18_finlit> 


Hi [first_name], using the <campus tutoring center name> math & writing 
center can help you do well on midterms & boost your grades before end 
of term. Can I help connect you? 


<R: tnec_f18_mathcenter> 


Hi [first_ name], make your tuition $$ count by taking advantage of 
academic resources like the <campus tutoring center name> math & 
writing center. Can I help connect you? 


<R: tnec_f18_mathcenter> 


Hi, priority registration for Spring 2019 starts today! Register before 
Sunday to maximize your chances of getting the classes you need to 
graduate. Can I help? 


<R: tnec_f18_registration> 
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November 5, 
2018 4:00 PM 


November 12, 
2018 4:00 PM 


November 19, 
2018 4:00 PM 


November 26, 
2018 4:00 PM 


November 26, 
2018 4:00 PM 


December 10, 
2018 4:00 PM 


December 19, 
2018 4:00 PM 


Hi [first_name], paying tuition on time guarantees your seat in the Spring 
2019 courses you need to graduate. If you already registered, last day to 
pay your tuition is Fri, 12/7. If you register after 12/7, tuition is due the 
day you register. Do you feel on track to pay on time? 


<R: tnec_f18_springpayment> 


Hi [first_ name]! Final exams are in | month. Many students use the 
<campus tutoring center name> math & writing center to prepare for 
exams & boost their GPA. Can I help connect you? 


<R: tnec_f18_academicresources> 


Hey, it’s <advisor name>. This week’s break from classes is a great time 
for you to get the FAFSA in for the 2019-2020 school year, if you haven’t 
already. 


<R: tnec_f18_fafsa_2> 


Hi [first_ name]! Only 2 weeks until final exams. A study plan can help 
you do well on finals & graduate on time. Can you set aside 2 hrs each 
day to study? 


<R: tnec_f18_studyplan> 


Hi [first_ name]! Only 2 weeks until final exams. A study plan can help 
you pass finals & advance towards a rewarding career. Can you set aside 
2 hrs each day to study? 


<R: tnec_f18_studyplan> 


Hi, you’re so close to passing your exams & moving closer toward your 
degree! The <campus tutoring center name> is open this week if you need 
tutoring. Can I help? 


Hi, congrats on moving closer toward your degree! Setting New Year 
goals can keep you on track to graduate on time. Which of these goals can 
you commit to? 


<R: tnec_f18 goals> 


34 


Appendix 2: Predictive Models of Dropout 


To examine which students with substantial credits stand to benefit from targeted outreach 
and support, we developed dropout prediction models at each partner institution using data on 
historical cohorts of students. We predicted the probability of dropout after students completed 30 
or 60 college-level credits at two- and four-year colleges, respectively, as a function of time- 
invariant student characteristics, measures of students’ enrollment experiences and performance 
in college, and measures of financial need and aid receipt. We then assigned risk ratings to students 
in the experimental sample using the dropout prediction models.'® We assigned risk ratings using 
logistic regression models in the pilot phase and random forest classification models during the 
scale phase. As we discuss in more detail below, the two models performed very similarly in 
absolute terms; however, we switched modeling strategies over time because the random forest 
models performed slightly better in relative terms. 

At each institution, we evaluated the performance of several candidate prediction models by 
splitting the historical data into development and validation samples to identify which model best 
distinguished between students who dropped out and students who graduated or were still enrolled 
in the historical data. The specific covariates included in each model differed slightly across 
institutions based on data availability. In general we compared the performance of models that 
only included predictors up to the term that students completed one-half of the credits typically 
required for graduation to models that also included measures of their enrollment history and aid 


receipt after completing one-half of their credits. The models that consistently performed best 


'8 Due to cost constraints and institutional preferences, recruitment was limited at some campuses. At institutions 
where the number of eligible students exceeded the number of recruitment slots, we also used the dropout predictions 
to exclude the most inframarginal students from the study sample. 
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captured information on students before and after they completed one-half of the credits typically 


required to graduate. These models include the following general set of predictors: 


1) 


2) 


3) 


Fixed student attributes and time-variant measures before students completed one-half of 
the credits typically required for graduation. Where available, this vector includes the 
following measures: age, gender, race/ethnicity, assignment to remediation status, whether 
the student transferred into their current institution and whether the student temporarily 
stopped out before completing one-half of their required credits to graduate. To capture 
changes in student circumstances over time that may influence risk of dropout, the vector 
also includes an indicator of whether students changed majors between when they first 
entered the institution (or system) and when they completed one-half of their credits, as 
well as within-student standard deviations of the following measures: Expected Family 
Contribution (EFC), the amount of financial aid received (entered separately by aid type), 
and the number of credits attempted per term. 

Measures of academic performance and financial aid receipt in the term students completed 
one-half of their credit requirements. Where available, this vector includes the number of 
attempted credits, cumulative GPA, the cumulative proportion of attempted credits that 
were earned, and the amount of financial aid received. 

Measures of enrollment experiences and financial aid receipt after surpassing the one-half 
credit threshold analogous to those captured in 1) above. 


Our preferred models effectively differentiated between late dropouts and non-late 


dropouts in the historical samples. For example, the probability that a randomly chosen late 


dropout was assigned a higher risk rating than a randomly chosen student who did not drop out 
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ranged from 0.75-0.875 across the models.'? Students in the experimental sample who graduated 
were also at lower risk of dropout on average compared to students who did not graduate. The 
average predicted probability of dropout was 23.2 percent among students who graduated 
compared to 38.3 percent among students who did not graduate. 

In Appendix Table A2, we report descriptive statistics for the full study sample by tercile 
of predicted dropout risk. The average risk rating in the bottom, middle, and top tercile is 0.13, 
0.26, and 0.50, respectively. Students at greatest risk of dropout exhibited higher rates of course 
failure and erratic credit loads as they progressed in school. For example, bottom-tercile students 
completed 96 percent of their attempted credits prior to intervention launch, whereas top-tercile 
students completed 85 percent of their attempted credits. High-risk students were also more likely 


to be older, male, and identify as Black or Latinx. 


'? On average, replacing logistic regression models with random forest algorithms increased the probability that a 
randomly selected late dropout was assigned a higher risk rating than a randomly selected non-dropout by less than 3 
percentage points (2-3 percent). The risk ratings generated by the two modeling approaches correlate around 0.90 or 
higher. 
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Appendix Table Al: N2FL Staffing Models 


Example Advisor Role Sample Message Institutions 
Advisor 
Background(s) 


Professional | Hired Direct assistance Hi, it’s <Professional PVCC, 
Advisor specifically for | with tasks (e.g., Advisor>. With finals TNCC, SPS, 
project registering for coming up, I wanted to JIC, LAGCC, 
courses, financial aid | check if you’ve used ACC, 
applications) <Support Center> for help | Lehman, 
with classes. CanI help |WCC, Alamo 
you get connected? 


Direct assistance Hey, it’s <Faculty 

with questions in Advisor>. As you’re 
their specialization |planning for spring, think 
(e.g., course about picking up an extra 
selection) and course. This can help you 
recommending graduate sooner. Can I 
campus resources for |help you choose another 
other questions (e.g., |class? 

financial aid) 


Staff Point |Administrative | Direct students to Hi <Student>! JTCC, 
Person assistant on the resource most Registration for falland |UTPB, Blinn, 
student appropriate for summer starts 4/2. Have |MECC, 
engagement providing assistance |you talked to an advisor |CVCC, UT 
team about the next classes you | Arlington 
need to take in your 
program? 


Mix of campus |Leveraged multiple | Hi, it’s <Advisor>. 

staff (e.g., some |staff depending on | Fafsa.gov is now open for 

faculty advisors | question (e.g., the 2018-2019 school year 

coupled with a_|student replies to and applying early gets 

career services |automated questions | you the most financial aid. 

counselor) about course Have you started FAFSA 
registration went to |yet? [student replies are 
an Academic routed to a Financial 
Advisor’s portfolio) |Advisor’s inbox] 
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Appendix Table A2. Pre-treatment characteristics of experimental sample by tercile 


of predicted dropout risk 


(1) (2) (3) 

poten Dede Top Tercile 

Tercile Tercile 
Male 0.412 0.027*** 0.038*** 
Black 0.116 0.030*** 0.070*** 
Hispanic 0.207 0.010 0.020*** 
White 0.420 -0.015** -0.049*** 
Race other 0.150 -0.027*** -0.039*** 
Race missing 0.106 0.001 -0.002 
Age 20.59 1.141*** 153370" 
Enrolled in public 2-year institutions 0.482 0.024*** Q:0377** 
Enrolled in public 4-year institution 0.518 -0.024*** -0.037*** 
Cumulative credits earned before 
intervention 65.50 -2.640*** -4.456*** 
Share of credits earned before intervention 0.957 -0.034*** -0.110*** 
Transferred into current school 0.292 -0.002 0.001 
Predicted risk of dropout 0.130 OL 238*"* O37 Ltte 
Number of Students: 7,192 7,183 7,178 


##* H<0.01 ** p<0.05 * p<0.10 


Notes: Notes: Means are reported in column 1. Differences relative to bottom-tercile students are 
reported in columns 2 and 3 from OLS/LPM models. Estimates include school by cohort fixed 
effects. Risk terciles are defined within a school. The respective means by tercile are 0.13, 0.26, 


and 0.50. 
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Appendix Table A3. Pre-treatment characteristics of VCCS experimental sample by 
treatment condition 


(1) (2) (3) 


Experimental Sample 


VCCS Treated VCCS Control VCCS T-C 


Students Students Difference 

Male 0.431 0.428 0.003 
Black 0.182 0.183 -0.001 
Hispanic 0.093 0.102 -0.008 
White 0.621 0.614 0.007 
Race other 0.039 0.044 -0.004 
Race missing 0.064 0.057 0.007 
Age 19.97 19.73 0.247 
Enrolled in public 2-year institutions 1.000 1.000 -- 
Enrolled in public 4-year institution 0.000 0.000 -- 
Cumulative credits earned before 

intervention 35.49 35.52 -0.001 
Share of credits earned before intervention 0.856 0.856 0.001 
Transferred into current school 0.109 0.108 0.002 
Predicted risk of dropout 0.459 0.458 0.001 
P-value on F-test for joint significance 0.837 
Number of Students: 2,889 1,447 4,336 


*** <0.01 ** p<0.05 * p<0.10 
Notes: The data in columns 1-3 are from VCCS partner institution administrative records. Means are 


reported in columns | and 2. Estimates of post-randomization balance are reported in column 3 from 
OLS/LPM models that include randomization block fixed effects. 
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Appendix Table A4. Pre-treatment characteristics of CUNY experimental sample 
by treatment condition 


(1) (2) (3) 
Experimental Sample 
Treated Control. CUNY TC 
Students Students PCECHCe 

Male 0.378 0.376 0.005 
Black 0.209 0.202 0.006 
Hispanic 0.281 0.287 -0.008 
White 0.174 0.176 0.000 
Race other 0.180 0.172 0.008 
Race missing 0.157 0.163 -0.005 
Age 2227 22.06 0.123 
Enrolled in public 2-year institutions 0.441 0.453 0.000 
Enrolled in public 4-year institution 0.559 0.547 0.000 
Cumulative credits earned before 

intervention 60.88 60.57 -0.023 
Share of credits earned before intervention 0.933 0.931 0.001 
Transferred into current school 0.306 0.309 -0.010 
Predicted risk of dropout 0.245 0.247 0.0004* 
P-value on F-test for joint significance 0.805 
Number of Students: 5,219 2,532 7,751 


*** <0.01 ** p<0.05 * p<0.10 

Notes: The data in columns 1-3 are from CUNY partner institution administrative records. Means 
are reported in columns | and 2. Estimates of post-randomization balance are reported in column 
3 from OLS/LPM models that include randomization block fixed effects. 


Appendix Table A5. Pre-treatment characteristics of TX experimental sample by treatment 
condition 


(1) (2) (3) 
Experimental Sample 
TX Treated TX Control TX T-C 
Students Students Difference 

Male 0.456 0.476 -0.020 
Black 0.082 0.090 -0.008 
Hispanic 0.317 0.307 0.009 
White 0.381 0.379 0.002 
Race other 0.104 0.104 0.000 
Race missing 0.116 0.120 -0.004 
Age 21.84 21.88 -0.045 
Enrolled in public 2-year institutions 0.550 0.550 0.000 
Enrolled in public 4-year institution 0.450 0.450 0.000 
Cumulative credits earned before 

intervention 61.55 62.25 -0.678** 
Share of credits earned before intervention 0.890 0.892 -0.002 
Transferred into current school 0.347 0.358 -0.011 
Predicted risk of dropout 0.262 0.262 0.000 
P-value on F-test for joint significance 0.413 
Number of Students: 3,690 1,845 5,535 


*** <0.01 ** p<0.05 * p<0.10 

Notes: The data in columns 1-3 are from TX partner institution administrative records. Means are reported 
in columns | and 2. Estimates of post-randomization balance are reported in column 3 from OLS/LPM 
models that include randomization block fixed effects. 
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Appendix Table A6: Estimates of intervention effects on academic outcomes by 
system, six term outcomes 


(1) (2) (3) (4) 
Re-Enrolled © Number of 
or Credits Earned 


Transferred to Four- 


Graduated, Accumulated, Degree, Six ; 
; ; Year, Six Terms after 
Six Terms Six Terms Terms after i 
: Intervention 
after after Intervention 
Intervention Intervention 
Panel A: VCCS Institutions 
Treatment 
Impact 0.004 0.013 -0.004 0.013 
(0.014) (0.608) (0.014) (0.018) 
Control Mean 0.756 24.868 0.682 0.396 
Observations 3095 3095 3095 3095 
Panel B: CUNY Institutions 
Treatment 
Impact -.015 -1.004 -.019 -.032** 
(0.011) (0.665) (0.012) (0.016) 
Control Mean 801 36.671 721 635 
Observations 5353 5353 5353 3693 
Panel C: TX Institutions 
Treatment 
Impact 0.002 2.840* -0.025 
(0.034) (1.570) (0.037) 
Control Mean 0.838 29.039 0.820 
Observations 500 500 500 


*** <0.01 ** p<0.05 * p<0.10 

Estimates are from OLS/LPM models that include risk rating, randomization block fixed 
effects, and the following pre- treatment covariates: indicators for sex, race/ethnicity (Black, 
Hispanic, Other, and Missing Race), and transfer status at the start fall 2016, as well as 
continuous measures of cumulative credits completed, and the fraction of total credits 
attempted that were earned at the start of the intervention. Robust standard errors are reported 
in parentheses. Column 4 only includes students at 2-year colleges. 
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Appendix Table A7. Estimates of intervention effects on academic outcomes by 
predicted dropout risk and system, four terms after intervention 


(1) (2) (3) (4) 
Re-Enrolled Number of aed Tiadetoed 
on Credits Degree to Four- 
Graduated, Accumulated, eres 
Four Terms Year, Four 
Four Terms Four Terms 
after Terms after 


after after 


. ; Intervention Intervention 
Intervention Intervention 


Panel A: VCCS Institutions 


Treatment x Bottom Tercile -.011 954 -.008 -.03 
(0.020) (0.714) (0.025) (0.027) 
Treatment x Middle Tercile .004 .678 -.011 029 
(0.026) (0.798) (0.027) (0.026) 
Treatment x Top Tercile -.009 263 -.02 014 
(0.028) (0.870) (0.024) (0.021) 
P-value on F-test of Equal 
Effects 895 828 935 Zoo 
Control Mean - Bottom Tercile 843 24.855 .687 489 
Control Mean - Middle Tercile .676 23,351 48 306 
Control Mean - Top Tercile 499 20.335 .263 18 
Observations 4336 4336 4336 4336 
Panel B: CUNY Institutions 
Treatment x Bottom Tercile 014 Le .003 -.041 
(0.014) (0.702) (0.017) (0.026) 
Treatment x Middle Tercile -.001 -.906 -.016 001 
(0.016) (0.708) (0.019) (0.025) 
Treatment x Top Tercile -.007 -.799 -.012 -.023 
(0.019) (0.724) (0.019) (0.026) 
P-value on F-test of Equal 
Effects 612 396 711 5 
Control Mean - Bottom Tercile 864 36.141 woe .698 
Control Mean - Middle Tercile 824 32.976 658 596 
Control Mean - Top Tercile .667 27.187 A491 A497 
Observations 7751 7751 7751 4064 


44 


Appendix Table A7, continued. Estimates of intervention effects on academic outcomes 
by predicted dropout risk and system, four terms after intervention 


(1) (2) (3) (4) 
Re-Enrolled Number of Pamed en eee 
ar Credits Degree to Four- 
Graduated, Accumulated, a 
Four Terms Year, Four 
Four Terms Four Terms 
after Terms after 
after after : : 
. F Intervention Intervention 
Intervention Intervention 
Panel C: TX Institutions 
Treatment x Bottom Tercile .024 1.051 024 
(0.018) (0.927) (0.018) 
Treatment x Middle Tercile -.004 954 O11 
(0.020) (0.923) (0.019) 
Treatment x Top Tercile 014 92 -.02 
(0.021) (0.928) (0.018) 
P-value on F-test of Equal 
Effects 588 995 201 
Control Mean - Bottom Tercile .767 27.161 489 
Control Mean - Middle Tercile .707 26.436 436 
Control Mean - Top Tercile .636 23,398 412 
Observations 5535 5535 5535 


##* H<0.01 ** p<0.05 * p<0.10 


Notes: Estimates are from OLS/LPM models that include risk rating, randomization block fixed 
effects, and the following pre- treatment covariates: indicators for sex, race/ethnicity (Black, 
Hispanic, Other, and Missing Race), and transfer status at the start fall 2016, as well as continuous 
measures of cumulative credits completed, and the fraction of total credits attempted that were earned 
at the start of the intervention. Robust standard errors are reported in parentheses. Risk ratings 
terciles are defined within an institution. Column 4 only includes students at 2-year colleges. 
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Appendix Table A8. Estimates of intervention effects (four terms after intervention) 


by school level 
(1) (2) (3) 
Re-Enrolled or Number of 
Graduated, Credits Earned Degree, 
Four Terms Accumulated, | Four Terms after 
after Four Terms after Intervention 
Intervention Intervention 
Treatment x 2 Year .00098 411 -.01 
(0.009) (0.372) (0.009) 
Treatment x 4 Year O11 43 .0045 
(0.007) (0.372) (0.008) 
P-value on F-test of Equal Effects 389 971 Bee ae) 
Control Mean - 2 Year .669 24.957 433 
Control Mean - 4 Year 844 43.475 73 
Observations 21553 21553 21553 


*#E <0.01 ** p<0.05 * p<0.10 

Notes: Estimates are from OLS/LPM models that include risk rating, randomization block fixed 
effects, and the following pre- treatment covariates: indicators for sex, race/ethnicity (Black, 
Hispanic, Other, and Missing Race), and transfer status at the start fall 2016, as well as continuous 
measures of cumulative credits completed, and the fraction of total credits attempted that were 
earned at the start of the intervention. Robust standard errors are reported in parentheses. 
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Appendix Table A9. Estimates of intervention effects (four terms after intervention) by advising 
model and institutional characteristics 


(1) (2) (3) 
Re-Enrolled Number of 
Credits Earned 
or Graduated, 
Accumulated, Degree, Four 
Four Terms 
Four Terms Terms after 
after : 
: after Intervention 
Intervention : 
Intervention 
Panel A: Impacts by Advising Model 
Treatment x Professional Advisor .017 821 -.003 
(0.015) (0.711) (0.015) 
Treatment x Team or Segmented Advisors .002 -.112 -.001 
(0.018) (0.793) (0.019) 
Treatment x Connectors -.003 .193 -.006 
(0.009) (0.330) (0.009) 
Treatment x Automated Advising 012 .673 .001 
(0.009) (0.535) (0.010) 
P-value on F-test of Equal Effects 589 714 .976 
Control Mean - Professional Advisor 584 25.736 343 
Control Mean - Team or Segmented Advisors .724 31.415 soil 
Control Mean - Connectors .784 27.96 595 
Control Mean - Automated Advising 842 48.08 74 
Observations 21553 21553 21553 
Panel B: Impacts by Institutional Characteristics 
Student: Counselor Ratio: 
Treatment x Stu:Counselor Ratio < 250:1 .013 .693 001 
(0.010) (0.427) (0.011) 
Treatment x Stu:Counselor Ratio > 250:1 -.003 -.082 -.01 
(0.008) (0.369) (0.008) 
P-value on F-test of Equal Effects 199 17 436 
Control Mean - Stu:Counselor Ratio < 250:1 .786 36.239 63 
Control Mean - Stu:Counselor Ratio > 250:1 .784 37.386 657 
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Appendix Table A9, continued. Estimates of intervention effects (four terms after intervention) 
by advising model and institutional characteristics 


(1) (2) (3) 
Re-Enrolled = Number of 
or Credits Earned 
Graduated, Accumulated, Degree, Four 
Four Terms Four Terms Terms after 
after after Intervention 
Intervention Intervention 
Other Texting Programs: 
Treatment x No Other Texting Programs 0065 nT .0046 
(0.009) (0.499) (0.010) 
Treatment x Other Texting Programs .00029 Ale -.014 
(0.009) (0.318) (0.009) 
P-value on F-test of Equal Effects 614 865 N62 
Control Mean - No Other Texting Programs 838 48.807 lz 
Control Mean - Other Texting Programs .738 26.429 589 
Required Advising: 
Treatment x No Required Advising 0 .03 -.009 
(0.011) (0.511) (0.011) 
Treatment x Required Advising .004 27 -.005 
(0.008) (0.325) (0.008) 
P-value on F-test of Equal Effects od .623 781 
Control Mean - No Required Advising 751 41.333 wd 
Control Mean - Required Advising 807 34.159 .695 
Student Support (composite measure from survey 
response) 
Treatment x Below Median Student Support .006 326 -.001 
(0.007) (0.352) (0.008) 
Treatment x Above Median Student Support .006 Pow ks -.006 
(0.009) (0.386) (0.010) 
P-value on F-test of Equal Effects 993 .634 .703 
Control Mean - Below Median Student Support .736 34.756 22 
Control Mean - Above Median Student Support .803 34.928 .698 
Observations 21553 21553 21553 


##* H<0.01 ** p<0.05 * p<0.10 
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Appendix Table A9, continued. Estimates of intervention effects (four terms after 
intervention) by advising model and institutional characteristics 


Notes: Estimates are from OLS/LPM models that include risk rating, randomization block fixed effects, and 
the following pre-treatment covariates: indicators for sex, race/ethnicity (Black, Hispanic, Other, and 
Missing Race), and transfer status at the start fall 2016, as well as continuous measures of cumulative 
credits completed, and the fraction of total credits attempted that were earned at the start of the intervention. 
Robust standard errors are reported in parentheses. Panel B reports effects based on responses to a survey 
sent to partner institutions in July 2020; representatives from 16 institutions responded to the survey (4 
institutions did not respond). Students from the institutions that did not respond to the survey are excluded 
from Panel B. The composite measure of student support is based on a survey question which asked whether 
the following services were offered and how proactively they were offered: academic tutoring; assistance 
with course selection; assistance with transferring to a 4-year university; assistance with completing the 
FAFSA; emergency financial aid; other financial assistance (resolving financial holds, checking aid status); 
assistance with food or housing; and career exploration. The three possible responses and how they were 
coded numerically were: this was not offered (point value of 0); this was offered, students must seek this 
support/service out on their own (point value of 1); this was offered, campus delivered this support/service 
proactively to students (point value of 2). The point value of each of those responses was averaged at the 
institution level. The median of that mean measure of student support was then calculated across institutions 
and institutions were identified as having above or below median student support based on that number. 
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Appendix Figure A1. Estimates of intervention effects by quartile of advisor responsiveness 
A. Re-Enrolled or Graduated, Four Terms after Intervention 
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Appendix Figure Al, continued. Estimates of intervention effects by quartile of advisor 
responsiveness 


C. Earned Degree, Four Terms after Intervention 


| 
| 
Quartile 1 |. ———- _--—_ 


Quartile 274 


Quartile 3 74 


f 


Quartile 47 


Notes: Each line reports coefficient estimates and 95% ClIs from OLS/LPM models within each quartile 
group that include risk rating, randomization block fixed effects, and the following pre- treatment 
covariates: indicators for sex, race/ethnicity (Black, Hispanic, Other, and Missing Race), and transfer 
status at the start fall 2016, as well as continuous measures of cumulative credits completed, and the 
fraction of total credits attempted that were earned at the start of the intervention. Advisor 
responsiveness is measured as the share of student text messages that an advisor responded to at the 
institution by cohort level. Quartile 1 is the lowest advisor responsiveness (mean .44) and Quartile 4 is 
the highest advisor responsiveness (mean .82). Students from the pilot phase (launch Fall 2016) are 
excluded from this analysis as we do not have data on advisor responsiveness for that cohort. 
Additionally, students from OU and UWT are removed from this analysis as advisor responses were 
automated for those institutions. 
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